This looks to be the bug @kwitaszczyk was running into, and the cause of the problem aligns with what @bsdjhb and I had managed to ascertain at the PI meeting.

The handling of vbar_el1 also looks a bit dodgy. If CPACR_EL1.CEN[0] is 0 then CVBAR_EL1 is just interpreted as VBAR_EL1 by the architecture when trapping to EL1 (setting PCC's address to it), which means we need to derive a capability for tf_elr from elr_el1.

The handling of vbar_el1 also looks a bit dodgy. If CPACR_EL1.CEN[0] is 0 then CVBAR_EL1 is just interpreted as VBAR_EL1 by the architecture when trapping to EL1 (setting PCC's address to it), which means we need to derive a capability for tf_elr from elr_el1.

This should be testable by running GDB against a plain FreeBSD VM, setting a breakpoint from GDB and triggering a breakpoint from within the VM, just as for the SPSR C64 issue except with a FreeBSD guest. I'm not sure how exactly I expect it to break, whether it'll get stuck in a trap loop or end up in bhyve. Hopefully at least it doesn't wedge the host, which should be true as long as some of this code is preemptible... otherwise there are other ways a malicious guest could trigger the same kinds of issues even with correct handling here.

This looks to be the bug @kwitaszczyk was running into, and the cause of the problem aligns with what @bsdjhb and I had managed to ascertain at the PI meeting.

Unfortunately, it doesn't seem the host kernel enters the block of if (hypctx->has_exception) in my case. It does enter it but once the guest kernel ends up in kdb_enter() after the panic.

Can you add a trace to see if vmmops_setreg is ever called to write a value to VM_REG_GUEST_CPSR?

Can you add a trace to see if vmmops_setreg is ever called to write a value to VM_REG_GUEST_CPSR?

It doesn't seem vmmops_setreg() is ever called for this purpose. I've added

diff --git a/sys/arm64/vmm/vmm_arm64.c b/sys/arm64/vmm/vmm_arm64.c
index a66d5ed1ba97..23efd3e4887d 100644
--- a/sys/arm64/vmm/vmm_arm64.c
+++ b/sys/arm64/vmm/vmm_arm64.c
@@ -1447,6 +1447,12 @@ vmmops_setreg(void *vcpui, int reg, uintcap_t val)
 #endif
                *(uintcap_t *)regp = val;
                break;
+       case VM_REG_GUEST_CPSR:
+               printf("%s:%d\nval=%lu\nelr_el1=%#lp\ntf_elr=%#lp\nspsr=0x%lx\n",
+                   __func__, __LINE__, (uint64_t)val, (void *)hypctx->elr_el1,
+                   (void *)hypctx->tf.tf_elr, hypctx->tf.tf_spsr);
+               *(uint64_t *)regp = (uint64_t)val;
+               break;
        default:
                *(uint64_t *)regp = (uint64_t)val;
                break;

and I don't get anything in the serial console.

I do think there are other things to check (CEN) that Jess noted, but this is certainly an improvement over what is there now.

I have a patch to address that comment, but had been holding off on pushing it until I could test with hybrid kernels. Now I'm looking at an apparent regression with the VHE merge after I rebased onto dev; hopefully it won't take too long to fix.